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Abstract 

Prevention scientists recognize that implementing effective prevention practices and programs responsive to the needs of 
individuals but based solely upon the findings from variable-centered methods presents several limitations due to numerous risk 
factors, pathways, and unobserved influences. One such understudied influence that is masked by variable-centered methods, 
motivation, is a person-level characteristic that influences treatment outcomes. The purpose of this paper is to demonstrate the use 
of an alternative person-centered approach, group iterative multiple model estimation (GIMME), to model change over time that 
focuses on the interdependence of daily student motivation levels and teacher feedback and their relations to student outcomes 
over time. Specifically, we used GIMME to model person level responses to negative teacher feedback regarding students’ next 
day motivational ratings using data from 58 Sth grade students participating in a study of the impact of the self-monitoring and 
regulation training strategy (SMARTS). Results identified a set of SMARTS students whose daily readiness aligned with high 
rates of self and teacher agreement regarding ongoing performance ratings. However, results identified a group of students whose 
daily motivation and readiness for change was adversely impacted by negative teacher feedback the day before. For these 
students, they were more likely than their peers to experience high levels of depression and internalization scores. 
Motivationally oriented practice suggestions for providing feedback to students who may be sensitive to this type of feedback 
and research implications of these findings are discussed. 
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Prevention scientists have long recognized implementing ef- 
fective prevention practices, and programs responsive to the 
needs of users but based solely upon the findings from 
variable-centered methods presents several limitations. First, 
due to numerous risk factors and pathways that interact with 
individual characteristics and contextual influences to condi- 
tion a person’s response to a treatment program, variable- 
centered methods mask our capacity to detect nuanced differ- 
ences in individual responses to any particular prevention pro- 
gram. This is a problem because no single program adequately 
targets all possible mechanisms in an individually tailored 
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manner. Second, the effect of motivation on treatment 
outcomes—a person-level characteristic—is widely 
understudied, particularly from a person-centered approach 
(Bergman and Magnusson 1997; Wiedermann et al. 2016). 
Even if the effect of individual motivation on outcomes was 
examined, the true effect of motivation on treatment uptake 
would likely remain masked in variable-centered analyses ex- 
amining moderators. The purpose of this paper is to demon- 
strate the use of an alternative person-centered approach to 
model change over time that focuses on the interdependence 
of daily student motivation levels and teacher feedback and 
their relations to student outcomes over time. 

With the dissemination of successful prevention programs, 
there has also been a growth in the use of prevention science 
technologies in the daily practices of professionals working 
with youth in school settings. For example, there is a growing 
use of individual risk screening and assessment tools and pro- 
cedures to identify youth with varying levels of risk (Dowdy 
et al. 2010). There is also increasing wide-spread adoption of 
tiered prevention and intervention frameworks which 
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integrate a range of programs and practices in an effort to 
combat an array of risk factors experienced at varying degrees 
by individuals in a population of youth (e.g., Positive 
Behavior Interventions and Supports; Flannery et al. 2014). 
Beyond prevention science technologies, there is also an in- 
creasing awareness of the influence of varying levels of indi- 
vidual motivation and how that predicts the success of preven- 
tion programs aiming to reduce aggression and peer victimi- 
zation (Cunningham et al. 2012) or improve academic moti- 
vation (Wigfield and Cambria 2012). 

Motivation or readiness to change is a common topic in 
studies examining precursors for change in the adult literature. 
Miller and Rollnick (1991) defined motivation as “the proba- 
bility that a person will enter into, continue, and adhere to a 
specific change strategy.” Motivation in this view is measured 
by asking the individual to report their readiness to engage in a 
behavior change action plan. This conceptualization of moti- 
vation helps explain heterogeneity of individual behavioral 
enactments as well as response to treatments. In line with this 
definition, Lopez-Viets et al. (2002) specified six key charac- 
teristics of motivation: (1) it is modifiable, (2) the relation 
between motivation and action is characterized by a probabil- 
ity, (3) motivation is influenced by the context in which it is 
discussed, (4) motivation is specific to each intended course of 
action and thus may vary based on the specific behavior being 
considered, (5) motivation is intrinsic and extrinsic, and (6) 
intrinsic motivation is more likely to increase when it is elic- 
ited from the individual rather than others. 

Motivational interviewing (MI; Miller and Rollnick 2012) 
is a clinical technique based on this definition and these prin- 
ciples of motivation and is a widely applied and evidence- 
based approach to absolving ambivalence and evoking readi- 
ness to change. Helpers use statements conditioned to elicit 
client change talk—the primary mechanism leading to behav- 
ior change in MI (McNamara 2009; Miller and Rose 2009). 
One statement MI helpers use is to provide personalized feed- 
back to a client about ongoing performance related to desir- 
able outcomes. The purpose of personalized feedback is to (a) 
verify whether a gap exists between current and expected 
performance levels, (b) ascertain the magnitude of the gap, 
(c) elaborate strategies to reduce the gap, and (d) assist with 
monitoring ongoing performance (Shute 2008). As one exam- 
ple applied in an educational context, Reinke and colleagues 
developed the MI-based classroom check-up (CCU) to give 
teachers specific feedback about specific domains of effective 
classroom management (e.g., rate of praise versus reprimands, 
rate of opportunities to respond; Reinke et al. 2011). 
Consultants present feedback to evoke change talk about what 
the teacher wants to be different. Several studies indicate that 
CCU improves use of best teaching practices (Reinke et al. 
2008; Reinke et al. 2011). 

Although MI informed that approaches have been less 
commonly used in interventions for youth, similar principles 
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apply (Herman et al. 2013). Many educational interventions 
employ personalized feedback to monitor and foster student 
progress in acquiring math and reading skills (Deno et al. 
2009; Fuchs et al. 2007). Reading interventions commonly 
use daily reading probes of performance skills like number 
of words-read-per-minute, and this data is charted and present- 
ed to the student (Fuchs and Fuchs 1999). Check in/check out 
(Crone et al. 2010) is a widely used tier 2 intervention that 
provides students with teacher daily ratings of student behav- 
iors that are targeted for improvement. Thus, formative feed- 
backs are staples of educational interventions targeting both 
academic and social-behavioral outcomes with much litera- 
ture supporting their effectiveness (Shute 2008; Reinke et al. 
2008; Reinke et al. 2007). 

An untested assumption of performance feedback is that all 
youth benefit from this type of feedback equally. Additionally, 
although performance feedback in educational contexts is 
consistent with MI, most existing educational applications of 
feedback are not informed by MI but are often driven by a 
behavioral orientation that hypothesizes that feedback in- 
creases awareness which by itself leads to be behavior change. 
Because feedback in school settings is nearly always provided 
by adults to students, it may be that the way adults present the 
feedback influences how it is received or that student charac- 
teristics influence the uptake of feedback. For instance, MI 
theory suggests that motivation is influenced by the interper- 
sonal context in which it is discussed. Thus, although person- 
alized feedback can evoke higher levels of motivation when 
delivered in a positive and supportive context, the opposite 
effect can occur when feedback is delivered in a judgmental 
and non-supportive context the probability of behavior 
change. 

An additional factor that may influence youth response 
to feedback is their pre-existing self perceptions of their 
competence and skills. MI theory predicts that self per- 
ceptions of competence may influence an individual’s re- 
sponse to feedback as well as their motivation or readi- 
ness to change a behavior. Evidence suggests that some 
youth with challenging behaviors, including both exter- 
nalizing and internalizing behaviors (Kaiser and 
Ramninsky 2009), hold inaccurate perceptions of perfor- 
mance (see Hoza et al. 2004; Owens et al. 2007). For 
instance, a significant body of research has found that 
youth with challenging behaviors hold what is known as 
a positive illusory bias (Hoza et al. 2004) where they 
overestimate their competence compared with the ratings 
by others. Unlike modest illusory biases observed in the 
general population, the bias exhibited by youth with chal- 
lenging behaviors is much more extreme and does not 
appear to foster or promote motivation to change. 
Although many explanations for the origins of this illuso- 
ry bias have been offered (immaturity, skill deficit), evi- 
dence suggests that the bias serves a self-protective 
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purpose where youth with challenging behaviors overes- 
timate ability to protect their social and self image 
(Owens et al. 2007). 

Given these biases and purported mechanisms, it is impor- 
tant for feedback interventions for youth with challenging 
behavior problems to examine variation in treatment re- 
sponses. Feedback which provides youth with information 
about discrepancies or agreement between their ratings of per- 
formance compared with others’ ratings could evoke positive 
or negative effects on youth motivation and subsequent per- 
formance dependent upon individual characteristics. 
Performance feedback is intended to narrow the discrepancy 
but could serve to undermine subsequent performance and 
motivation in youth threatened by the feedback’s unmasking 
of their illusory bias self-image protection. One intervention, 
the self-monitoring and regulation training strategy 
(SMARTS), is guided by the principles of motivational 
interviewing and provides performance and discrepancy feed- 
back to students with challenging behaviors. 


SMARTS 


The SMARTS (formerly called STARS; Thompson 2014; 
Thompson and Webber 2010) is a manualized tier 2 support 
intervention for youth with challenging behaviors that in- 
cludes many of the elements of performance feedback. 
SMARTS includes features of existing selective programs that 
use formative adult feedback to drive behavioral change (1.e., 
Check, Connect, and Expect; The Behavior Education 
Program, check in/check out), but SMARTS also attempts to 
promote autonomy supportive opportunities for students to 
self monitor their own goal performance and compare their 
own perspectives with that of teachers. SMARTS structures 
direct instruction in key social-emotional skills to shape stu- 
dent goal formation, self monitoring, and processing of dis- 
crepancy feedback. Following student training, SMARTS 
structures daily opportunities for students to monitor social- 
emotional goals, compare self and teacher performance feed- 
back, and consider strategies to reduce discrepancies between 
self and teacher perspectives. Self monitoring is a widely used 
motivational support strategy. Theoretically, self-monitoring 
is an autonomy support activity which is a key element in 
self-determination theory (Deci and Ryan 2011). When edu- 
cators use autonomy support strategies like self monitoring, 
they promote student choice, directly involve students, in- 
crease opportunities for students to practice, and structure 
more opportunities for educators to clarify expectations for 
classroom behavior (Wigfield et al. 2008; Wentzel 2002) 
which has been shown to directly increase motivation to per- 
form the expected behaviors via the concepts of engagement 
and reactivity (Pintrich & Schunk, 2002). Reactivity is a 
metacognitive principle referring to behavioral change 


occurring as a function of observing one’s own behavior. 
Observed in studies of self monitoring of learning, students 
who self monitor math performance gain increased awareness 
of the number of problems they have correct and engage in 
strategies that directly improve performance (Bandura 2005; 
Cleary and Zimmerman 2004). The reactivity principle has 
also been observed in self-monitoring as an intervention in 
weight loss and substance abuse studies (Boutelle and 
Kirschenbaum 2012; Butryn et al. 2012; Sinadinovic et al. 
2010). Targeted interventions like SMARTS that focus on 
tapping motivational strategies and utilize performance feed- 
back to encourage students to learn, adopt, and utilize adaptive 
behaviors at school are individualized approaches to behavior 
support. Individualized approaches in motivation are useful in 
that it permits school personnel to tailor goals and feedback to 
meet student needs. However, these individualized interven- 
tions also present opportunities for alternative methods for 
analyzing change. 

The purpose of our paper is to discuss group iterative mul- 
tiple model estimation (GIMME; Gates and Molenaar 2012; 
Beltz and Gates 2017) as a subject-specific approach to eval- 
uate dynamic relations in intensive longitudinal data. GIMME 
uses structural equation models (SEMs) that account for se- 
quential dependence observed in time series data to identify 
population-, shared-, and individual-level relations among 
variables. The present study uses GIMME to evaluate the 
dynamic relation between daily teacher feedback on student 
behavioral goal performance and student motivation and its 
impact on student mental health outcomes. We hypothesized 
that multiple causal classification patterns would emerge from 
the analyses including subgroups of students for whom feed- 
back affected subsequent student motivation and others for 
whom motivation affected subsequent feedback. 
Additionally, we examined the classification patterns’ relation 
to student outcomes including student-reported externalizing 
and internalizing symptoms. We hypothesized that negative 
classification patterns, that is, those marked by a decline in 
student motivation or an increase in negative feedback, would 
predict worse student emotional outcomes. Finally, we deter- 
mined the relations between baseline student-teacher relations 
and classification patterns and hypothesized that negative 
baseline student-teacher relations would predict negative clas- 
sification patterns. 


Method 


The sample used in the present study was collected as part of 
an ongoing randomized control study to evaluate SMARTS. 
Two hundred ninety-three students were recruited for partici- 
pation in the study using a dual-gated screening and assess- 
ment system. The first gate relied on the behavioral and emo- 
tional screening system (BESS; Kamphaus and Reynolds 
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2007) to screen 100% (N=2059) of 5th grade students. 
Students identified as high risk on the BESS then were 
assessed at the second gate using the behavioral assessment 
system for children (Reynolds et al. 2011). Gate 2 inclusion 
criteria required students to exhibit a T score at or above 60 on 
the externalizing, internalizing, or school problems scales. As 
such, the present study included 154 students with elevated 
risk across three cohorts (2016: n =49 students, 2017: n=59 
students, 2018: n = 46 students, see online supplement) who 
were randomly assigned to the SMARTS intervention only 
(i.e., control condition students are not included). Twenty- 
three students (14.9%) were excluded due to not meeting the 
gate 2 T score requirement. 

Of the remaining 131 students (11.1 years ofage; SD = 0.4) 
in the sample, 73.3% qualified for free or reduced lunch and 
self-identified as White (43.5%) Black (41.2%), Biracial 
(8.4%), Hispanic (4.6%), or Asian (2.3%). The sample includ- 
ed more male than female students (71.8%) and 25.8% re- 
ceived a special education service. Students showed average 
internalizing and externalizing T scores of 64.8 (SD = 15.0) 
and 72.6 (SD = 14.9), respectively (further descriptive statis- 
tics are given in the online supplement). Following Wright 
et al. (2019) and Lane et al. (2019), 72 students (55.0%) 
who had more than 40% missing values in daily self monitor- 
ing and teacher feedback on individual goals were discarded 
from this study. A cutoff of > 40% was a reasonable trade-off 
between convergence rates of individual time series models 
and retaining as many students as possible for subsequent 
(variable oriented) analyses of students’ mental health out- 
comes. In addition, one student reported no variability in daily 
motivation and was excluded, leaving this analysis to be con- 
ducted with a sample of n = 58 students. Students in the anal- 
ysis sample were less likely to qualify for free or reduced 
lunch (63.8%) than students in the subsample with > 40% 
incomplete data (80.8%, p = 0.047) and tended to show lower 
baseline depression (analysis sample M = 68.0, SD = 14.7; ex- 
cluded subsample M=72.9, SD=15.7; p=0.071) and 
follow-up externalization T scores (analysis sample M= 
66.1, SD = 14.3; excluded subsample M= 70.7, SD= 15.5; 
p=0.086). No significant differences between the analytical 
sample and excluded subsample were observed for age, race, 
gender, special education service, cohort, baseline externaliz- 
ing and internalizing T scores, and follow-up internalizing and 
depression T scores. Table | (third column) gives the descrip- 
tive statistics for the sample. 


Intervention Procedures 


Delivery of SMARTS is organized in three phases: training, 
self monitoring and feedback, and processing. SMARTS 
training is delivered by student support personnel (school psy- 
chologists counselors, social workers) in a small group ses- 
sions lasting about 35-40 min each using ten scripted lessons 
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in: (1) pre-group meeting, (2) assessing and defining prob- 
lems, (3) generating and considering solutions, (4) writing 
measureable goals, (5) observing and recording progress, (6) 
using data to evaluate progress, (7) taking the perspective of 
others, (8) reframing mistakes as part of learning, (9) manag- 
ing internal responses to problems, and (10) managing exter- 
nal responses to problems. Following training, students begin 
daily self monitoring where both students and teachers mon- 
itor students’ performance each hour of the day on their indi- 
vidualized goals. Each day, however, before students and 
teachers rate ongoing goal performance, students respond to 
three motivational prompts scaled 0—“not at all” to 10— 
“very ready,” measuring (a) how rested they feel, (b) how 
positive their mood is, and (c) how ready they are to accom- 
plish their goal. Next, students and teachers select one of three 
response options (“yes,” “sometimes,” or “no’’) reflecting the 
student’s goal performance during the prior 1-h interval. 
Teachers’ recording of student behavior performance is simi- 
lar to other tier 2 approaches (i.e., CICO). Unique to SMARTS 
and not CICO, SMARTS students are trained in goal devel- 
opment and self-monitoring skills, they use a website to chart 
and record performance, and can view graphs and percentages 
of their self and teacher ratings of their progress. Lastly, phase 
3 consists of a weekly processing meeting between a school 
counselor and SMARTS students where they discuss in great- 
er detail the daily performance feedback collected over the 
week. Using the web-based dashboard percentages and 
graphics, SMARTS students compare self and teacher obser- 
vational data by responding to several motivational 
interviewing prompts encouraging students to compare pres- 
ent data with (a) their current goal, (b) prior goals and perfor- 
mance data, and (c) concurrent teacher data. During process- 
ing, students also consider behaviors that contributed to the 
discrepancies. Using the data and behavioral information, stu- 
dents then revise their goals each week. The revised goal is 
then entered into the website using observable and 
measureable language, and the student and teacher perfor- 
mance monitoring and processing are repeated iteratively. 
All three phases were implemented similarly and at the same 
time during the school year across each of the three cohorts. 
Median time series lengths were 32 (Q; = 30, Q3 = 32) out of 
32 possible days in 2016, 39.5 (OQ, = 33.8, O03; =41) out of 41 
possible days in 2017, and 38 (Q; = Q3 = 38) out of 42 possi- 
ble days in 2018. 


Measures 


Following gate one screening, consent/assent, and gate two 
pretests, students were randomly assigned to SMARTS or a 
control. Posttest surveys were completed following SMARTS 
training and 8 weeks of self and teacher monitoring. In the 
analysis, we used daily student readiness ratings and daily rate 
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Table 1 Demographic 


information and descriptive Variable 

statistics of the analysis sample 
Female n(%) 
Black n(%) 
FRL n(%) 
SPED n(%) 
Cohort n(%) 
2016 
2017 
2018 
Age (in years) M (SD) 
NTF M (SD) 
Student motivation M (SD) 
Externalization (pre) M (SD) 
Internalization (pre) M (SD) 
Depression (pre) M (SD) 
Externalization (post) M (SD) 
Internalization (post) M (SD) 
Depression (post) M (SD) 


NTF — SM SM — NTF or no Total(n = 58) 
(n=12) relation (n = 46) 

3 (25.0) 11 (23.9) 14 (24.1) 
5 (41.7) 17 (37.0) 22 (37.9) 
10 (83.3) 27 (58.7) 37 (63.8) 
4 (33.3) 11 (23.9) 15 (25.9) 
6 (50.0) 13 (28.3) 19 (32.8) 
5 (41.7) 13 (28.3) 18 (31.0) 
1 (8.3) 20 (43.5) 21 (36.2) 
11.2 (0.4) 11.1 (0.5) 11.1 (0.5) 
0.61 (0.44) 0.47 (0.42) 0.50 (0.43) 
8.46 (2.91) 8.31 (2.37) 8.34 (2.49) 
70.3 (10.1) 71.5 (13.1) 71.2 (12.5) 
67.4 (13.3) 61.1 (13.9) 62.4 (13.9) 
69.3 (14.4) 67.7 (15.0) 68.0 (14.7) 
69.0 (15.0) 65.4 (14.2) 66.1 (14.3) 
69.0 (10.1) 57.9 (13.8) 60.2 (13.8) 
74.3 (17.2) 63.0 (15.4) 65.3 (16.4) 


M, mean; SD, standard deviation; pre, pre-treatment measure; post, post-treatment measure; F'RL, free/reduced 
lunch; SPED, special education service; NTF, proportion of daily negative teacher feedback 


of negative teacher feedback as predictors of social-emotional 
outcomes and student rated relations with their teacher. 


Student Daily Readiness 


Student participants rated daily readiness for change using 
three MI rulers patterned after Miller and Rollnick (1991) 
asking “How positive do you feel today?”’, “How well rested 
do you feel today?”, and “How ready are you to follow your 
goal today?” with responses ranging from 0 (low readiness) to 
10 (high readiness). Similar to studies using MI rulers to score 
daily readiness for change, the three items here were com- 
bined into a single composite (a = 0.86) representing daily 
readiness for change. Readiness for change rulers are brief 
measures of a person’s motivation for change and are highly 
correlated with multi-item readiness questionnaires measuring 
readiness for change in studies of alcohol and substance use 
treatment (0.77; LaBrie et al. 2005), smoking cessation (0.87; 
Boudreaux et al. 2012), and safe sex prevention practices 
(0.77; LaBrie et al. 2005). 


Negative Teacher Feedback 


Negative teacher feedback was measured as the daily relative 
frequency of cases where a positive performance rating was 
observed by the student but a negative performance rating was 
entered by the teacher (i.e., the student and teacher disagreed 
on whether the student achieved the goal). Frequencies ranged 


from 0 to 1 with higher scores indicating more greater rates of 
daily negative teacher feedback. 


Social-Emotional Health Outcomes 


Student social-emotional health was measured using the be- 
havior assessment system for children 3—teacher rating sys- 
tem (BASC; Reynolds et al. 2011). Specifically, we relied on 
BASC subscales that measured internalization problems (a = 
0.85— 0.89) and externalization problems (a= 0.83 —0.86). 
We also used the BASC 3 self-report of personality to ask 
youth about their depressive symptoms (« = 0.93). 


Student-Teacher Relations 


Students responded to the teachers who care scale of the ele- 
mentary school success profile (ESSP-Student; 5 items, a = 
0.72; Bowen 2010) measuring perceptions of whether a teach- 
er listened, praised, provided help, and got along with the 
student. 


Analysis 


We used GIMME to evaluate longitudinal patterns of associ- 
ations of negative teacher feedback and student motivation. 
GIMME is a data-driven approach that allows to distinguish 
population, shared, and individual effects from intensive lon- 
gitudinal data. Population effects are conceptualized as effects 
that occur for the majority of individuals comprising the 
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sample (effects are allowed to vary across individuals). Shared 
effects are those that occur for subgroups of the sample, and 
individual effects reflect processes that are unique to the given 
individual. 

The GIMME algorithm is based on a unified structural 
equation modeling (uSEM; Kim et al. 2007; Lane et al. 
2019; Molenaar 2019) framework linking standard SEMs 
and vector autoregressive (VAR) models (Liitkepohl 2005) 
and provides lagged and concurrent effects of negative teacher 
feedback and student motivation. Lagged effects refer to cases 
where measures of one construct (e.g., teacher feedback) at 
time point ¢ predicts another construct (student motivation) at 
the subsequent time point ¢+ 1. Concurrent effects describe 
direct effects of one construct on the other the same day 
(e.g., negative teacher feedback affects student motivation at 
time ¢) and occur when the underlying data-generating mech- 
anism changes faster than the rate of data collection (Granger 
1969). In addition, autoregressive effects are estimated which 
quantify how well a variable predicts itself over time. Such 
autoregressive effects can be conceptualized as measures of 
stability. Positive autoregressive effects reflect the extent to 
which a given measure at time point f+ 1 can be predicted 
by the prior measure of the same construct at time point ¢ 
negative effects are usually conceptualized as a feedback sys- 
tem whereby the system cycles between low and high values 
(cf. Wright et al. 2019). Figure 1 summarizes all possible SEM 
paths for negative teacher feedback (NTF) and student moti- 
vation (SM). 

GIMME starts with estimating an empty model across each 
individual, and SEM modification indices are used to indicate 
the anticipated improvement in model fit. To distinguish be- 
tween sample, shared, and individual effects, we used previ- 
ously validated cutoff values which have been shown to reli- 
ably recover true effects (Gates and Molenaar 2012; Lane 
et al. 2019). To identify population effects, we used a cutoff 


a 
NTF) | ———> | NTF 
1) (2. py Db, ©l) |S 
a 
SM, | ———> | SMaay 


Fig. 1 First-order lagged structural model to evaluate longitudinal 
association patterns of negative teacher feedback (NTF) and student mo- 
tivation (SM). The parameters a, and a denote autoregressive effects, by 
and b are cross-lagged effects, and c, and cz are contemporaneous effects 
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value of > 75%, that is, GIMME (iteratively) identifies paths 
that optimally improve the fit of the group model for more 
than 75% of the individuals until no further paths can be 
found. Then, the obtained group model is pruned by eliminat- 
ing those paths which, because of the freeing up of paths at 
later iterations, no longer are acceptable according to the 75% 
criterion (Gates and Molenaar 2012). Next, GIMME com- 
putes matrices based on group-level path estimates of pairs 
of individuals and uses a robust community detection algo- 
rithm known as Walktrap (Pons and Latapy 2006) to identify 
shared temporal effects (i.e., subgroup-level paths). We used a 
cutoff value of > 50% to identify subgroup-level paths, that is, 
a path had to significantly improve model fit for at least 50% 
of a subgroup to be identified as a shared effect. A subgroup- 
level pruning procedure was used, where paths were removed 
which are no longer significant for >50% of individuals. 
Finally, during the individual-level search, significant individ- 
ual paths (using a = 0.01 for testing the significance of mod- 
ification indices) were added until an “excellent” model fit 
was obtained, that is, when two out of the four model fit 
indices (root mean square error of approximation (RMSEA), 
non-normed fit index (NNFI), comparative fit index (CFD, 
standardized root mean square residual (SRMR)) suggested 
an excellent model fit (RMSEA <0 .05, SRMR <0.05, CFI 
> 0.95, NNFI > 0.95; Brown 2006). Because GIMME is lim- 
ited to first-order VARs, lagged effects beyond previous mea- 
surement occasions (such as NTF(t) ~ NTF(t+ 2) or 
SM(t) > SM(t+2)) were not considered. Time series data 
for teacher feedback and student motivation were de-trended 
prior model estimation (Lane et al. 2019). Full information 
maximum likelihood (FIML) estimation was used to accom- 
modate missing data. Instead of imputing missing values or 
conducting listwise deletion, FIML uses all of the available 
data to estimate model parameters (Enders 2010). All analyses 
were performed in R (R Core Team 2019) using the package 
gimme (Lane and Gates 2017). 


Results 
GIMME Results 


As expected, the two autoregressive paths NTF(t) — NTF(t+ 
1) and SM(4) > SM(t+ 1) were identified as population ef- 
fects and freely estimated for each individual. We observed a 
median regression weight of — 0.01 (Q; =— 0.18, Q3 =0.15) 
for NTF(Q) > NTF(¢+ 1) and — 0.03 (Q; =— 0.16, Q3 = 0.21) 
for SM(t) > SM(t+1). Only 48% of the estimated 
autoregressive effects of NTF and 47% of autoregressive ef- 
fects of SM were positive indicating relative inconsistency 
across days. In the present sample, negative teacher feedback 
and student motivation can be characterized as feedback sys- 
tems cycling between low and high values. 
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No subgroup (shared) effects were identified using a 50% 
cutoff value. Instead of further lowering the subgroup detec- 
tion cutoff, we focused on a substantive classification of 
individual-level effects that were detected for 24 students 
(41.4%; for the remaining 34 students only autoregressive 
effects, NTF(t) ~ NTF(¢+ 1) and SM(t) > SM(t+ 1), were 
needed). Individual-level effects confirmed that student moti- 
vation and negative teacher feedback influence each other 
over time, but there is considerable heterogeneity in these 
carryover effects (cf. Fig. 2). Inspecting individual-level 
models reveal interesting longitudinal dynamics. Some indi- 
viduals have lagged and/or contemporaneous causal links be- 
tween NTF and SM, but the paths can go into opposite causal 
directions. For example, 7 individuals (12.1%) had a contem- 
poraneous effect of the form SM(‘) > NTF(f) (Fig. 2, lower 
right panel) and 8 individuals (13.8%) showed a causally re- 
versed effect NTF(‘) — SM(f) (Fig. 2, lower left panel). 
Further, 14 individuals (24.1%) had lagged effects SM(t) > 
NTF(¢+ 1) (Fig. 2, upper right panel) and only 7 individuals 
(12.1%) showed causally reversed lagged effects of the form 
NTF) — SM(¢+ 1) (Fig. 2, upper left panel) suggesting that, 
in the present sample, student motivation is more likely to 


affect negative teacher feedback than vice versa. While high 
heterogeneity is observed for all individual-level effects, neg- 
ative teacher feedback tends to increase student motivation on 
the same day but lowers student motivation on the subsequent 
day. 


Student Mental Health Outcomes 


Next, we evaluated the predictive power of individual causal 
classifications (NTF — SM, SM > NTF, or independence of 
NTF and SM) for student post-treatment mental health. While 
SM — NTF constitutes the dominant causal mechanism in the 
present sample, we focused on the reverse causal model; stu- 
dents are affected by negative teacher feedback. Overall,12 
students (20.7%) either had lagged or contemporaneous ef- 
fects of the form NTF — SM. Figure 3 (online supplement) 
gives the individual-level models for the 12 students and 
Table 1 shows descriptive statistics for pre- and post- 
treatment measures of teacher-rated externalization and inter- 
nalization problems, and depression. Table 2 gives the results 
of linear regression models for mental health outcomes 
adjusting for gender (1 = female, 0 = male), race (1 = black, 
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Table 2 Multiple linear 
regression results for examining 
the effects of being influenced by 


Internalization problems 
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negative teacher feedback b SE 

(NTF — SM) on post-treatment 

internalization problems, exter- NTF — SM 6.96 3.81 

nalization problems, and Baseline measure 0.46 0.12 

ea Female 0.92 3.41 
Black -1.90 3.58 
FRL 3.04 3.46 
SPED 6.40 3.51 
Age 0.32 3.47 
Cohort: 2017” -2.83 4.16 
2018 -—0.90 3.97 
R 0.48 


Externalization problems Depression 
72) b SE Pp b SE 72) 
0.074 3.69 3.21 0.256 9.40 3.80 0.017 
< 0.001 0.89 0.12 <0.001 0.71 0.12 <0.001 
0.788 1.23 2.88 0.672 2.49 3.40 0.468 
0.597 -1.57 3.00 0.603 -447 3.54 0.212 
0.383 2.02 3.16 0.525 2.03 3.55 0.571 
0.075 0.42 2.78 0.881 2.84 3.61 0.436 
0.926 3.67 2.93 0.217 0.03 3.48 0.993 
0.500 —2.95 3.55 0.410 —3.29 424 0.442 
0.821 -1.77 3.41 0.606 —-162 406 0.692 

0.65 0.63 


b, unstandardized regression coefficient; SE, standard error; p = p value; FRL, free/reduced lunch; SPED, special 


education service 


*2016 cohort is used as the reference 


0 = other), age (in years), free/reduced lunch (1 = yes, 0 = no), 
special education (1 = yes, 0=no), study cohort, and pretest 
mental health measures. Students whose motivation is affect- 
ed by negative teacher feedback tend to be have more inter- 
nalization problems (5 = 7.0, p = 0.074, d= 0.24) and are more 
depressed (b= 9.4, p=0.017, d=0.32) at the end of the se- 
mester compared with students who are not affected by teach- 
er feedback. There were no significant effects for externaliza- 
tion problems. 

Finally, we examined whether baseline student-teacher re- 
lations differentially predicted the individual causal classifica- 
tions (NTF — SM, SM > NTF, or independence of NTF and 
SM). Overall, there are no significant differences in the per- 
ceived student-teacher relation for the two GIMME groups 
independent of measurement occasion (pre and post interven- 
tion) and rater (student and teacher). All models were adjusted 
for the covariates listed above. 


Discussion 


We used a small subset of students participating in a study of 
SMARTS, a self-monitoring training program, to examine 
temporal dynamics between student daily ratings for their 
readiness or motivation for change and rates of agreement 
between students and teacher perceptions of goal performance 
feedback. Following rigorous screening and inclusion criteria, 
58 students provided daily readiness for change ratings. These 
ratings were used alongside daily self and teacher ratings of 
goal performance to identify groups of individuals who ap- 
peared to fit into several patterns. Noteworthy among these 
patterns are subgroups of students where daily readiness rat- 
ings influenced subsequent daily performance as well as a 
group of students where negative teacher feedback (i.e., larger 
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discrepancies between student self ratings and teacher ratings) 
adversely impacted subsequent readiness. 

For most students, daily readiness begets improved align- 
ment between student self and teacher ratings of goal perfor- 
mance. However, for a small subset of youth, it appears that 
daily readiness has an unintended or iatrogenic effect on rates 
of student and teacher agreement rates. One explanation of 
these unintended consequences is that for these students, high 
levels of daily readiness may contribute to them feeling over- 
confident and exacerbate pre-existing illusory biases. These 
students may benefit from daily reminders or prompts to assist 
them in minimizing or avoiding the overconfidence trap or 
overinflated perceptions of their own self and their perfor- 
mance abilities (Hoza et al. 2004; Owens et al. 2007). 

A second subgroup effect that was observed is that if the less 
common causal relation NTF(4) > SM(¢+ 1) exists for a student, 
there tends to be a negative relation (6 out of 7 effects were 
negative). In other words, for this subset of student, increased 
disagreement led to lowered readiness for change the next day— 
and in turn may have the impact of disengaging a student from an 
intervention where formative feedback is provided frequently. In 
short, and contrary to the literature on formative daily feedback, 
some students may require that feedback be tailored in certain 
ways that optimizes their capacity to internalize that feedback 
and contribute to positive outcomes for that student. If the pur- 
pose of ongoing performance feedback is to (a) verify whether a 
gap exists between current and expected performance levels, (b) 
learn the magnitude of that gap, (c) elaborate strategies to reduce 
the gap, and (d) monitor the effectiveness of those strategies 
(Shute 2008), then the findings here suggest that some students 
may require tailored feedback in ways that they can take advan- 
tage of the feedback. The manner that feedback is discussed or 
presented to students may be a parallel mechanism worth explor- 
ing. In this study, although we provided training and support for 
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school personnel in presenting feedback, we were measuring 
variation in the presentation style of feedback discussions. 
Consistent with MI, the language adults use in discussing feed- 
back with youth will influence subsequent motivation. The find- 
ings imply that more rigorous training and/or more structured 
feedback scripts and protocols may be helpful in minimizing 
the number of students who present this iatrogenic pattern. 

Studies suggest teachers using task-specific and construc- 
tive feedback can positively influence student behaviors 
(Sutherland et al. 2000); however, findings here imply some 
students experience feedback differently. Our findings suggest 
that some students may be vulnerable to negative feedback in 
that it interferes with subsequent motivation and may exacer- 
bate depressive symptoms. Depressive symptoms are com- 
monly rooted in interpretative biases and distortions 
(Herman et al. 2013); thus, it is possible that students vulner- 
able to this effect may have cognitive predispositions to inter- 
nalize negative feedback as evidence of failure or disappoint- 
ment. Future research will need to include baseline measures 
of cognitive biases to test whether pre-existing cognitive style 
predicts vulnerability to iatrogenic response to feedback; if so, 
such students may benefit from cognitive restructuring inter- 
ventions and/or feedback may need to include direct conver- 
sations to interpret negative feedback in more constructive 
ways. 

That student-rated student-teacher relations did not predict 
classification groupings was surprising. This may indicate that 
the negative impact of teacher feedback the prior day on student 
readiness the next day is not due to preexisting differences in 
student-teacher relations but developed independently. The find- 
ing is consistent with the idea that feedback operates at the person 
level (1.e., internalized) so that it is not contingent on prior or 
subsequent relationship status. From an MI perspective, one pos- 
sibility is that the more proximal influence over motivation is the 
way the feedback is presented by teachers to students may elicit 
sustained talk and that feedback leads students to be less 
motivated—and in turn more depressed or less happy. 

There are noteworthy limitations to the study that we will 
attempt to address here. First and foremost, the analysis here 
relied on a subsample of a larger study. There is no control 
group in this study. The GIMME results presented here and 
the groupings are idiographic and relative to these students, in 
this study, using these measures. Further, intensive longitudi- 
nal data are known to be prone to missingness. The present 
results are based on 58 students (45% of the total sample), and 
we cannot rule out potential selection biases. The number of 
available time points per student was rather small (median 
time series length of 37.5; QO; =32.0, O3=38.0) compared 
with previous applications of GIMME. Using Monte Carlo 
simulations, Lane et al. (2019) showed that, for 30 observa- 
tions per subject and 5 variables, GIMME achieves a path 
recovery of 82.4%. Path recovery tends to increase with small- 
er numbers of variables, thus we can expect adequate path 


recovery for 2 variables (student motivation and teacher feed- 
back). Although we were able to detect population- and 
individual-level effects, the present study may have been un- 
derpowered to rule out shared (sub-group) effects. We also 
recognize that there are many factors at play in shaping any- 
one’s motivation—and very few of these factors are captured 
here. For example, some kids getting feedback about discrep- 
ancy may undermine readiness. In particular, we did not mon- 
itor the manner in which students were provided feedback. It 
is entirely reasonable to assume that a teacher who is exposed 
to MI strategies and uses the concepts of motivation to frame 
communications with students will have an entirely different 
outcome. Though this assumption can be measured and 
tested—and it should be—it also implies something should 
be done to reduce negative effects and reiterates our point; 
not all persons receive feedback in the same manner. If a 
student is involved in a daily ongoing performance feedback 
intervention, and there is a discrepancy between student and 
teacher views, then we need to be mindful about how that 
discrepancy is discussed. One possible factor not represented 
in these analyses is that we did not capture the ability of 
teachers and student support personnel to give feedback con- 
sistent with the values and themes of MI. Such training is 
vital—so is measuring the capacity to provide feedback pro- 
moting youth to engage in adaptive actions. Lastly, though we 
relied on similar but adapted (i.e., changed wording to read 
“ready” instead of “‘motivated”) measures to reflect the con- 
ceptualization of readiness as prior studies have done (Miller 
and Rollnick 1991), we recognize that these measures do not 
fully capture the complex nature of motivation. 

Future applications of GIMME hold promise for unpacking 
person-specific treatment responses and guiding improvements 
in preventive interventions that minimize iatrogenic effects. 
Such applications will require prevention studies to collect rich, 
repetitive data streams of key intervention processes most likely 
related to outcomes. In turn, advances in efficient, valid mea- 
sures that are sensitive to change and that natural implementers 
can consistently use as intended are needed to best capitalize on 
the benefits of analytic tools such as GIMME. 
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